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SYNTHESIZING COMPUTER INPUT 
EVENTS 

FIELD OF THE INVENTION 

This invention relates generally to computer input events, 
and more particularly to synthesizing such events. 

BACKGROUND OF THE INVENTION 

Computers today typically come with one or more input 
devices to enable users to input data into the computer. Such 
input devices include joysticks, mouses, and keyboards. An 
input event is thus a specific utilization of an input device. 
For example, moving the mouse, pressing a key on the 
keyboard, and clicking a button on the joystick, are all 
specific utilizations of input devices, and therefore are all 
input events. 

While the use of input devices for computers make them 
easier to use, they are still nevertheless not necessarily the 
most intuitive manner by which computer input can be 
achieved. For example, computer games in which a player 
must navigate through a three-dimensional (3D) terrain may 
have their life-like quality marred by the fact that the player 
must utilize a joystick or a mouse (i.e., two-dimensional 
devices) to signal to the computer in which direction move- 
ment in three dimensions is desired. In other words, the use 
of input devices such as joysticks, computer mouses and 
keyboards may not be the most natural user interface for 
achieving computer input. 

Therefore, there is a need for other methods by which 
computer input can be realized. Such other methods of input 
should be more intuitive than typical computer input devices 
allow. Perceptual user interfaces that user computer vision 
tracking of the user or users and speech recognition can 
provide for more intuitive computer input. 

SUMMARY OF THE INVENTION 

The invention relates to synthesizing computer input 
events. In one embodiment, a computerized system includes 
a sensor, a controller, a utility and an application. The sensor 
detects external events, such as audio events (e.g., speech or 
sounds as detected by a microphone) and video events (e.g., 
head movement of a user as detected by a video camera). 
The controller generates control variables in response to 
these external events. The utility then synthesizes input 
events, such as joystick, mouse, and keyboard events, from 
the control variables. The application thus utilizes these 
input events, as if they emanated from an input device such 
as a joystick, mouse or keyboard, itself. In addition to this 
embodiment, the invention includes other embodiments, 
including computerized systems, methods, computers, and 
computer-readable media of varying scope. 

BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 shows a block diagram of a computerized system 
in accordance with an embodiment of the invention; 

FIG. 2 shows a flowchart of a computerized method in 
accordance with an embodiment of the invention; and, 

FIG. 3 shows a diagram of a computer in conjunction with 
which embodiments of the invention may be practiced. 

DETAILED DESCRIPTION OF TOE 
INVENTION 

In the following detailed description of exemplary 
embodiments of the invention, reference is made to the 



)6,476 Bl 

2 

accompanying drawings which form a part hereof, and in 
which is shown by way of illustration specific exemplary 
embodiments in which the invention may be practiced. 
These embodiments are described in sufficient detail to 

5 enable those skilled in the art to practice the invention, and 
it is to be understood that other embodiments may be utilized 
and that logical, mechanical, electrical and other changes 
may be made without departing from the spirit or scope of 
the present invention. The following detailed description is, 

10 therefore, not to be taken in a limiting sense, and the scope 
of the present invention is defined only by the appended 
claims. 

Referring first to FIG. 1, a block diagram of a comput- 
erized system according to one embodiment of the invention 

15 is shown. The computerized system of FIG. 1 includes a 
sensor 100, a controller 102, a utility 104, a storage 106, an 
event queue 108 and an application 110. As shown, the 
controller 102, the utility 104, the storage 106, the event 
queue 108, and the application 110 are desirably part of a 

20 computer 112, although the invention is not so limited. In 
such an embodiment, the controller 102, the utility 104, and 
the application 110 are all desirably computer programs 
running on the computer 112 — i.e., programs (viz., a con- 
troller program, a utility program, and an application 

25 program) executed by a processor of the computer from a 
computer-readable medium such as a memory thereof. The 
computer 112 also desirably includes an operating system, 
not shown in FIG. 1, within which and in conjunction with 
which the programs run, as can be appreciated by those 

30 within the art. 

The sensor 100 detects external events, such as audio 
events and video events. In one embodiment, the sensor 100 
includes a microphone to detect audio events. These audio 
events may include spoken words, such as "move left," 

35 "move right, "stop," etc., or they may include sounds. In 
another embodiment, the sensor 100 includes a video cam- 
era to detect video events. These video events may include 
movement of an object, such as a computer user's head, 
tracked by the video camera. Such user movement that may 

40 be tracked by the video camera includes movement of the 
user's head in six degrees of freedom (X, Y, Z, roll, pitch and 
yaw), a human pose (full body or upper torso only), hand 
gestures, and facial gestures or emotions. Other objects that 
may be tracked include objects that are held in the user's 

45 hand or hands, such as an imitation gun, or a pencil. 

The sensor 100 is operatively coupled to the computer 112 
of which controller 102 is desirably a part. The invention is 
not limited to the manner by which the sensor 100 is so 

5Q operatively coupled, however. Desirably, the sensor 100 is 
operatively coupled via a hardware card installed within the 
computer 112, via a Universal Serial Bus (USB) within the 
computer 112, as known within the art, or via other manners. 
In response to the external events detected by the sensor 

55 100, the controller 102 generates control variables. That is, 
the controller 102 takes the raw data regarding the external 
events as detected by the sensor 100, and converts them to 
control variables reflecting the external events that the raw 
data represents. For example, in the case of the sensor 100 

60 including a microphone to detect audio events, the control 
variables may signify the words "Move Left" when the user 
has stated "Move Left," etc. As another example, in the case 
of the sensor 100 including a video camera to detect video 
events, the control variables may include the distance from 

65 a neutral position that the user's head has moved. 

The invention is not limited to any particular manner by 
which the controller 102 generates control variables from 
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the external events detected by the sensor 100. In the case of Thus, in the embodiment of the invention shown in FIG. 

audio events, speech recognition software, such as that 1, ultimately the application 110 receives and utilizes the 

readily available within the art, may be used. In the case of input events synthesized by the utility 104, either directly 

video events, the invention described in the coassigned, receiving it from the queue 108, or indirecdy through an 

cofiled, and copending application entitled "Control Vari- 5 operating system of the computer 112. Desirably, the appli- 

ables Generation for Computer Vision," docket no. cation 110 runs as a foreground application on the computer 

884.005US1, may be utilized, such that the controller 102 112j m conjunction with the operating system thereof. The 

corresponds to the driver 102 and the controller 104 of FIG. application 110 utilizes the input events synthesized by the 

1 of that application. utilily ^4 desirably as if they were "rear input events 

From the control variables generated by the controller 10 g enera ted in response to actuation of an input device such as 

102, the utility 104 synthesizes input events. Such input a joystick, mouse or keyboard. 

events include joystick events such as left, right, up, and Fof c fc m the case of an application 110 that is a 

down, mouse events, also such as left, right up, and down, uter the syDlhes i ze d input events may mimic the 

keyboard events such as various keys or key sequences eyents Qf fl . ^ m deyice me user states 

being pressed, and button events, such as one of the buttons 15 ^ kft ^ ^ 1Q4 ^ ^ . eveQt 

ofthe joystick or mouse being pressed. Desirably, the utility ^ ^ of ^ . ^ lQ ^ left> ^ if ^ 

104, in the case where it is a computer program running on . ick had acmaU beeQ mQved tQ ^ Jeft ^ , he 

the computer 112, is a background apphcation running in ^ Ucztion uo ma d lo me m t event b movi 

conjunction within an operalmg system of the computer 112. £ md of yiew tQ ^ fcft> ^ ^ case whefe me 

(7>ty1b determine the appropriate input events to synthesize 20 c0 uter game is a fli ght simu i ator . 

V for use by the application 110, as will be later described, the . . . . _ . . . . u _ _ t . 

fc-i-i-* h\a j • ui \u ♦ |A a—~~'~- As Das Deen described in conjunction with FIG. 1, the 

futility 104 desirably accesses the storage 106 to determine 4 * j-.u - 

,-*ri ' 4 • ■ . 1 . .. inn c u invention provides for advantages not found in the prior art. 

^parameters regarding the apphcation 100. Such parameters , -j * ■ ♦ u 

ffi&i . 1 j ,l . * * I*- ♦ !»• «• The mvention provides for a more intuitive manner by 

fcpinay mclude, in the context of a multi-tasking computing ... ** . ,. .... . ... • 

Sirrj • ' . ^ , _i- 1-fn ~f iii thS which user input is achieved, in the cases where utilizing 

^environment, the actual active application 11U or all the 2 s a l •_ . 

* . M , , voice commands, or where head movement, are a more easy 

applications runnmg on the computer 112, as can be appre- . . P ' . , . J 

jm~ • « 11 . • *• 1 ■ y „ -.u- ,u ^ Tn,^ ..t™^ and reahslic manner for the user to signify a desired action 

^ciated by those of ordinary skill within the art. The storage . & . J 

in , • 1 j 1 ■ 1 • „ju*u ~-~u „o than actuation of a computer input device. This makes for a 

106 may include a disk file in which such parameters are . . , J T_ . .. _ , , 

stored. In .he case of versions of the Microsoft™ Win- more n , atural user mterface ' enhancm S *» realism afforded 

dows™ operation system, the storage 106 may include the 30 by a PP hcatl0n P ro § rams runnm 8 Wllhm * computer, 

reaistrv ' " Referring next to FIG. 2, a flowchart of a computerized 

Furthermore, the utility 104 desirably is amenable to input me ' hod i° accordance with an embodiment of the invention 

events such as keyboari entry so that the user has a manner J 5 shown ' ™ e computerized method is desirably realized at 

by which to adjust parameters regarding the utility 104 itself. least 10 P a « one or more P ro S ram £ mnnlD g on a 

For example, in the case of video events, the user is 35 computer-that K , as a program executed from a computer- 

desirably able to manually adjust sensitivity and event rate, readable m ^ um ^ 35 a mem , or y ^ a P 100 ^ of a 

to re-zero the neutral position of the object tracked by the computer. The programs are desirably storable on a 

sensor 100, to pause and resume the synthesis of input ^P^TT? u f , nV T ^ ° f " 

events, and to exit the event synthesis, as can be appreciated CD-ROM, for ^distribution .and installation and execution on 

by those of ordinary skill within the art. In addition, since 40 anolher ^ lti ^ f^W*) computer The method of FIG. 

faster compute* typically respond to input events more \ 15 specifically the manner by which an external event is 

quickly than slower oomputerv-i.e., faster graphics render- de ' ected .> and an ln P ut event 18 synthesized based thereon 

ing rates result in faster apparent movement, and vice- utilized. 

versa— desirably the utility 104 is able to adjust itself to the In step 200, an external event is detected. The event may 
overall speed of the computer 112. For example, when the 45 be 311 audio event » such ^ a s P okea word or a ^und, or a 
utilily 104 is first run, the utility 104 may determine the v ^eo event, such as movement of an object like a user's 
speed ofthe computer 112, and adjust itself appropriately. In head or hand - ^ event is desirably detected by a sensor, 
some instances, more complex computer graphic scenes such as a microphone or a video camera, 
render slower than simpler graphic scenes, which results in In step 202, control variables are generated in response to 
slower or faster apparent movement, respectively. Desirably, 50 lh e external event. That is, the raw data representing the 
the utility 104 is able to also adjust for variations in graphic external event are translated lo control variables represent- 
rendering rates. ing the external event. For example, if the user's head is 
In the embodiment of the invention shown in FIG. 1, the moved to the left, the control variables representing this 
utility 104 enters the synthesized input event into the event external event may signify the distance the user's had has 
queue 108, just as other generated input events (i.e., as 55 moved relative to a neutral position. Desirably, the genera- 
generated by actual input devices such as keyboards, tion of the control variables is performed by a controller 
joysticks, and mouses) are inserted into the event queue 108 program running on a computer. 

by the operating system of the computer 112 (not shown in In step 204, an input event is synthesized based on the 

FIG. 1). In conjunction with the insertion of the synthesized external event. More specifically, the input event is gener- 

input event into the event queue 108, the utility 104 also 60 ated from the control variables generated in step 202. The 

desirably generates an interrupt, such as a hardware IRQ input event is desirably a joystick event, a mouse event, a 

(interrupt request) to signify that an input event has been keyboard event, or a button event. The input event thus 

placed in the queue 108. The event queue 108 itself may be mimics the actuation of a corresponding input device. For 

managed by the operating system of the computer 112. The example, in response to control variables representing the 

invention is not limited to the presence or absence of an 65 user having stated "Move Right" into the sensor, an input 

event queue 108, or whether or how an interrupt is event may be synthesized to represent as if a joystick input 

generated, however. device has actually been moved lo the right. Desirably, the 
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synthesis of the input event is performed by a utility program 
running on a computer in the background. 

Finally, in step 206, the input event is utilized. Desirably, 
the input event is utilized by an application program running 
on a computer in the foreground. For example, in response 
to an input event synthesized to represent as if ajoystick has 
been moved to the right, the field of view displayed by the 
application program may be panned to the right. Therefore, 
the synthesized input event is utilized as if the actual input 
event occurred. 

Desirably, the input event synthesized in step 204 is 
appropriate to the specific utilization of such an event in step 
206. This is accomplished by referencing parameters regard- 
ing the specific utilizations of input events, as such param- 
eters are stored in a storage such as a disk file or the registry 
in the case of the Microsoft™ Windows™ operating system. 
Furthermore, desirably, the input event is placed in an event 
queue that may be administered by the operating system of 
a computer, and an appropriate interrupt-such as a hardware 
IRQ — is correspondingly generated. 

Referring finally to FIG. 3, a diagram of a representative 
computer in conjunction with which embodiments of the 
invention may be practiced is shown. Computer 310 is 
operatively coupled to monitor 312, pointing device 314, 
and keyboard 316. Computer 310 includes a processor 
(desirably, an Intel™ Pentium™ processor), random-access 
memory (RAM) (desirably, at least thirty-two megabytes), 
read-only memory (ROM), and one or more storage devices, 
such as a hard disk drive, a floppy disk drive (into which a 
floppy disk can be inserted), an optical disk drive, and a tape 
cartridge drive. The memory, hard drives, floppy disks, etc., 
are types of computer-readable media. The invention is not 
particularly limited to any type of computer 310. Computer 
310 desirably is a PC-compatible computer, running a 
version of the Microsoft™ Windows™ operating system. 
The construction and operation of such computers are well 
known within the art. 

Monitor 312 permits the display of information within a 
viewing area, including computer, video and other 
information, for viewing by a user of the computer. The 
invention is not limited to any particular monitor 312, and 
monitor 312 is one type of display device that may be used 
by the invention. Such monitors include cathode ray tube 
(CRT) displays, as well as flat panel displays such as liquid 
crystal displays (LCD's). Pointing device 314 permits the 
control of the screen pointer provided by the graphical user 
interface of operating systems such as versions of 
Microsoft™ Windows™. The invention is not limited to any 
particular pointing device 314. Such pointing devices 
include mouses, touch pads, trackballs, wheels, remote 
controls and point sticks. Finally, keyboard 316 permits 
entry of textual information into computer 310, as known 
within the art, and the invention is not limited to any 
particular type of keyboard. 

Synthesis of computer input events have been described. 
In one embodiment, a computerized system includes a 
sensor, a controller, a utility and an application. The sensor 
detects external events, such as audio events and video 
events. The controller generates control variables in 
response to the external events. The utility synthesizes input 
events, such as joystick, mouse, and keyboard events, from 
the control variables. The application then utilizes these 
input events. 

Although specific embodiments have been illustrated and 
described herein, it will be appreciated by those of ordinary 
skill in the art that any arrangement which is calculated to 
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achieve the same purpose may be substituted for the specific 
embodiments shown. This application is intended to cover 
any adaptations or variations of the invention. It is mani- 
festly intended that this invention be limited only by the 
5 following claims and equivalents thereof. 
We claim: 

1. A computerized system comprising: 

a sensor detecting external events selected from the group 
essentially consisting of audio events and video events; 
30 a controller generating control variables in response to the 
external events; 

a utility synthesizing input events from the control 
variables, wherein the input events comprise events 
selected from the group essentially consisting of: joy- 
15 stick events, mouse events, keyboard events and button 
events; and, 

an application utilizing the input events. 

2. The computerized system of claim 1, wherein the utility 
accesses at least one of a disk file and a registry to determine 

20 parameters regarding the application to synthesize input 
events relevant to the application. 

3. The computerized system of claim 1, wherein the utility 
inserts the input events in an event queue for the application 
to utilize the input events. 

25 4. The computerized system of claim 1, wherein the utility 
generates an interrupt when synthesizing an input event. 

5. The computerized system of claim 1, wherein the utility 
runs as a background application running in conjunction 
with an operating system and the application runs as a 

30 foreground application running in conjunction with the 
operating system. 

6. The computerized system of claim 1, wherein at least 
one of the controller, the utility, and the application com- 
prises a computer program running on a computer. 

35 7. The computerized system of claim 1, wherein the 
sensor comprises a video camera detecting video events. 

8. The computerized system of claim 7, wherein the video 
events comprise movement of an object tracked by the video 
camera. 

40 9. The computerized system of claim 8, wherein the object 
comprises a head of a user. 

10. The computerized system of claim 1, wherein the 
sensor comprises a microphone detecting audio events. 

11. The computerized system of claim 10, wherein the 
45 audio events comprise spoken words detected by the micro- 
phone. 

12. The computerized system of claim 10, wherein the 
audio events comprise sounds detected by the microphone. 

13. A computerized system comprising: 

50 a sensor detecting external events selected from the group 
essentially consisting of audio events and video events; 
a controller generating control variables in response to the 
external events; 

55 a utility synthesizing from the control variables input 
events selected from the group essentially consisting 
of: joystick events, mouse events, keyboard events and 
button events; 

an application utilizing the input events in response to an 
50 interrupt generated by the utility when synthesizing an 
input event; 

a storage accessed by the utility to determine parameters 
regarding the application to synthesize input events 
relevant to the application; and, 
65 an event queue into which the input events synthesized by 
the utility are inserted by the utility for the application 
to utilize the input events. 
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14. The computerized system of claim 13, wherein the and video events, wherein the input events comprise 
storage comprises at least one of a disk file and a registry. events selected from the group essentially consisting 

15. A computerized method comprising: of: joystick events, mouse events keyboard events and 
detecting an external event selected from the group essen- button events. 

tially consisting of an audio event and a video event; 5 22. The computer of claim 21, wherein the utility program 

and, runs as a background application in conjunction with an 

synthesizing an input event based on the external event, operating system. 

the input event selected from the group essentially 23. The computer of claim 21, further comprising an 

consisting of: a joystick event, a mouse event, a key- application program executed by the processor from the 

board event and a button event. 10 medium to utilize the input events synthesized by the utility 

16. The computerized method of claim 15, further com- program. 

prising prior to synthesizing the input event, generating a 24. The computer of claim 23, wherein the application 

control variable in response to the external event. program runs as a foreground application in conjunction 

17. The computerized method of claim 16, wherein syn- ^th an operating system. 

thesizing the input event comprises generating the input 15 2 5. The computer of claim 21, further comprising a 

event from the control variable. controller program executed by the processor from the 

18. TTie computerized method of claim 15, further com- medium {Q ate lhe vafiables m re l0 the 
pnsmg utilizing the input event external events. 

19. The computerized method of claim 15, wherein the t, - , . , . . . 

* . , - u . ' 20 26. The computer of claim 21, wherein the external events 

video event comprises movement of an object. * ... 

20. The computerized method of claim 15, wherein the are detected by a sensor operatively coupled to the computer, 
audio event is selected from the group essentially consisting 27 Acomputer-readab le medium having a utility program 
of a spoken word and a sound. stored thereon for executl0n on a «"tobly equipped com- 

ii a « , • • . puter to synthesize input events from control variables 

21. A computer comprising. r _ / r . . . 

25 generated in response to external events selected from the 

a processor, group essentially consisting of audio events and video 

a computer-readable medium; and, events, wherein the input events comprise events selected 

a utility program executed by the processor from the from the group essentially consisting of: joystick events, 

medium to synthesize input events from control vari- mouse events, keyboard events and button events. 

ables generated in response to external events selected 30 

from the group essentially consisting of audio events * * * * * 
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